LCORE-2493: final touches for tokenizer benchmark by tisnik · Pull Request #1916 · lightspeed-core/lightspeed-stack

tisnik · 2026-06-12T06:35:02Z

Description

LCORE-2493: final touches for tokenizer benchmark

Type of change

Tools used to create PR

Assisted-by: N/A
Generated by: N/A

Related Tickets & Documents

Related Issue #LCORE-2493

Summary by CodeRabbit

Tests
- Extended benchmark suite with tests for larger input files (up to 10,000 lines) covering multiple formats: JSON, YAML, Python, JavaScript, XML, and text samples.
Documentation
- Updated and added benchmark performance reports reflecting expanded test coverage from 22 to 28 total tests, including new metrics for larger-scale inputs.

coderabbitai · 2026-06-12T06:35:15Z

Warning

Review limit reached

@tisnik, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 55 minutes and 39 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 011294da-3f1c-47a3-970e-48c7d9284682

📥 Commits

Reviewing files that changed from the base of the PR and between 2dea647 and 9d3dbf9.

📒 Files selected for processing (6)

tests/benchmarks/data/js_10000_lines.js
tests/benchmarks/data/json_10000_lines.json
tests/benchmarks/data/python_10000_lines.py
tests/benchmarks/data/python_1000_lines.py
tests/benchmarks/data/xml_10000_lines.xml
tests/benchmarks/data/yaml_10000_lines.yml

Walkthrough

This PR extends the tokenizer benchmarking suite by adding test coverage for larger input scales (10,000-line files and highly repeated text), executes those tests, and commits the resulting performance reports alongside minor cleanup.

Changes

Tokenizer benchmark expansion for larger inputs

Layer / File(s)	Summary
Benchmark test cases for larger inputs `tests/benchmarks/test_token_estimator.py`	Six new test functions benchmark tokenizer token estimation for JSON, YAML, JavaScript, Python, and XML files at 10,000 lines each. One additional test benchmarks lorem ipsum text repeated 2,000 times. All follow the existing `benchmark_file_tokenization()` or `benchmark(estimate_tokens, ...)` pattern.
Benchmark result reports `docs/benchmarks/tokenizer/10_lines.txt`, `docs/benchmarks/tokenizer/100_lines.txt`, `docs/benchmarks/tokenizer/1000_lines.txt`, `docs/benchmarks/tokenizer/10000_lines.txt`, `docs/benchmarks/tokenizer/all.txt`	Five benchmark result files document measured performance across all tokenizer tests. The summary reports timing and throughput metrics for 10, 100, 1,000, and 10,000-line scales, with `all.txt` consolidating results from all 28 tests executed in 23.43 seconds.
Test data cleanup `tests/benchmarks/data/python_1000_lines.py`	Removed a section comment following document ingestion and simplified the `print_rag_response()` helper by removing its docstring in favor of a brief inline comment.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

lightspeed-core/lightspeed-stack#1889: Extends tests/benchmarks/test_token_estimator.py with new benchmark test cases for token estimation at 100-line scale, following the same file-based fixture pattern.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	The title 'LCORE-2493: final touches for tokenizer benchmark' is vague and generic, using non-descriptive terms like 'final touches' that don't convey specific information about the changeset.	Consider a more specific title that describes the actual changes, such as 'Add tokenizer benchmark tests for 10,000-line inputs and update benchmark results' to better communicate the primary changes.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	Docstring coverage is 94.44% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

✨ Simplify code

Create PR with simplified code

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/benchmarks/data/python_1000_lines.py`:
- Around line 811-814: Restore a proper docstring for the public function
print_rag_response to satisfy pydocstyle D103: replace the inline comment with a
triple-quoted docstring immediately below the def print_rag_response(response):
line that succinctly describes the function's purpose (e.g., "Print a RAG-style
response to stdout"), documents the parameter `response` and its expected
type/format, and notes return behavior (usually None); ensure the docstring is a
top-level string under the def so linters recognize it.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: f35d68b2-a170-4087-aaa8-c5ad28724d84

📥 Commits

Reviewing files that changed from the base of the PR and between d447014 and 2dea647.

⛔ Files ignored due to path filters (5)

docs/benchmarks/tokenizer/10000_lines.svg is excluded by !**/*.svg
docs/benchmarks/tokenizer/1000_lines.svg is excluded by !**/*.svg
docs/benchmarks/tokenizer/100_lines.svg is excluded by !**/*.svg
docs/benchmarks/tokenizer/10_lines.svg is excluded by !**/*.svg
docs/benchmarks/tokenizer/all.svg is excluded by !**/*.svg

📒 Files selected for processing (12)

docs/benchmarks/tokenizer/10000_lines.txt
docs/benchmarks/tokenizer/1000_lines.txt
docs/benchmarks/tokenizer/100_lines.txt
docs/benchmarks/tokenizer/10_lines.txt
docs/benchmarks/tokenizer/all.txt
tests/benchmarks/data/js_10000_lines.js
tests/benchmarks/data/json_10000_lines.json
tests/benchmarks/data/python_10000_lines.py
tests/benchmarks/data/python_1000_lines.py
tests/benchmarks/data/xml_10000_lines.xml
tests/benchmarks/data/yaml_10000_lines.yml
tests/benchmarks/test_token_estimator.py

📜 Review details

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (17)

GitHub Check: radon
GitHub Check: ruff
GitHub Check: Pylinter
GitHub Check: Pyright
GitHub Check: bandit
GitHub Check: build-pr
GitHub Check: unit_tests (3.12)
GitHub Check: unit_tests (3.13)
GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-on-pull-request
GitHub Check: Konflux kflux-prd-rh02 / lightspeed-stack-0-6-on-pull-request
GitHub Check: E2E: library mode / ci / group 1
GitHub Check: E2E: library mode / ci / group 2
GitHub Check: E2E: server mode / ci / group 2
GitHub Check: E2E: server mode / ci / group 3
GitHub Check: E2E: library mode / ci / group 3
GitHub Check: E2E: server mode / ci / group 1
GitHub Check: E2E Tests for Lightspeed Evaluation job

🧰 Additional context used

📓 Path-based instructions (1)

tests/**/*.py

📄 CodeRabbit inference engine (AGENTS.md)

tests/**/*.py: Use pytest for all unit and integration tests; do not use unittest
Use pytest.mark.asyncio marker for async tests

Files:

tests/benchmarks/test_token_estimator.py
tests/benchmarks/data/python_1000_lines.py

🪛 GitHub Actions: Pydocstyle / 0_pydocstyle.txt

tests/benchmarks/data/python_1000_lines.py

[error] 814-814: pydocstyle (via uv tool run pydocstyle -v src tests) reported: D103: Missing docstring in public function print_rag_response.

🪛 GitHub Actions: Pydocstyle / pydocstyle

tests/benchmarks/data/python_1000_lines.py

[error] 814-814: pydocstyle: D103 Missing docstring in public function print_rag_response.

🔇 Additional comments (6)

tests/benchmarks/test_token_estimator.py (1)

126-138: LGTM!

Also applies to: 202-213, 258-269, 314-325, 370-381, 426-437

docs/benchmarks/tokenizer/10_lines.txt (1)

1-11: LGTM!

docs/benchmarks/tokenizer/100_lines.txt (1)

1-11: LGTM!

docs/benchmarks/tokenizer/1000_lines.txt (1)

1-11: LGTM!

docs/benchmarks/tokenizer/10000_lines.txt (1)

1-10: LGTM!

docs/benchmarks/tokenizer/all.txt (1)

1-37: LGTM!

tisnik added 2 commits June 12, 2026 08:32

Benchmark results

6358705

Benchmarks for large sets with 10000 lines

39297d9

New data sets

9d3dbf9

tisnik force-pushed the lcore-2493-final-touches-for-benchmark branch from 2dea647 to 9d3dbf9 Compare June 12, 2026 06:37

coderabbitai Bot reviewed Jun 12, 2026

View reviewed changes

Comment thread tests/benchmarks/data/python_1000_lines.py Outdated

tisnik merged commit 525e34e into lightspeed-core:main Jun 12, 2026
32 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LCORE-2493: final touches for tokenizer benchmark#1916

LCORE-2493: final touches for tokenizer benchmark#1916
tisnik merged 3 commits into
lightspeed-core:mainfrom
tisnik:lcore-2493-final-touches-for-benchmark

tisnik commented Jun 12, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 12, 2026 •

edited

Loading

Review limit reached

❌ Failed checks (1 inconclusive)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tisnik commented Jun 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Tools used to create PR

Related Tickets & Documents

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

❌ Failed checks (1 inconclusive)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tisnik commented Jun 12, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 12, 2026 •

edited

Loading